Robotic Weed Removal System for Aesthetic Mulch Gardens

ABSTRACT

An apparatus and method for removing weeds from an aesthetic mulch garden using an autonomous battery-powered differential-wheeled robot is disclosed. The boundary of the domain of said aesthetic mulch garden is predefined by the user and said robot is confined to patrol within said domain. The robot searches for weeds using machine vision which seeks colorimetric contrast between weeds and mulch and undergoes a novel randomized reflective trajectory to patrol said domain. An independent collision avoidance system allows said robot to avoid interaction with non-weed objects. Said robot has a central processing unit (CPU) receiving input from said machine vision system which positions a device for weed extraction and controls said extraction. A built-in suction system and receptacle is incorporated in said robot to maintain the weed extraction device clean and ready for operation while storing extracted weeds for later disposal.

FIELD OF THE INVENTION

Embodiments of the invention described herein pertain to the field ofweed removal from earth by an autonomous differential wheeled robot.More particularly, but not by way of limitation, embodiments of theinvention enable a robot system and method of robotic weed removal inaesthetic mulch gardens.

BACKGROUND OF THE INVENTION

In a comprehensive review by Slaughter et al. (Elsevier, 2007), it isstated that “Autonomous robotic weed control systems hold promise towardthe automation of one of agriculture's few remaining unmechanized anddrudging tasks, hand weed control. Robotic technology may also provide ameans of reducing agriculture's current dependency on herbicides,improving its sustainability, and reducing its environmental impact.”This 2007 review describes three basic components needed in anagricultural robot: “(1) a sensing system to measure important physicaland biological properties of the agricultural system (GPS, machinevision, imaging); (2) decision-making capabilities for processinginformation from the sensor system to determine how the agriculturalsystem should be manipulated (CPU); and (3) actuators to manipulate theagricultural system accordingly (chemical spray, cutting, thermal,electrocution).” Slaughter discusses extensively the need for plantspecies identification by means of machine vision so as to discriminatebetween weeds, crops, and other plants. He states that there are manystudies of ground-based machine vision plant species recognition. Thetechnology used in agricultural applications is extremely sophisticatedand requires complex sensing equipment. A prior art agriculturalautonomous intra-row weed control system described by Slaughter is shownin FIG. 1 and a schematic of a typical prior art weed control robottrajectory using intra-row guidance is shown in FIG. 2 . The articlepoints out that while herbicide-based weed control may be bothbiologically and economically effective, it has environment costs due torunoff and contamination of rivers and wetlands. In many locations,increasing pesticide use regulations, consumer concerns, and growinginterest in organically produced foods limit the long-term applicationof herbicide application. Slaughter teaches that by virtue of precisioncontrol, robotic systems can eliminate or reduce the quantity ofherbicide needed for agricultural weed control. However, the systems hedescribes are highly complex and require large robots as suggested bythe fact that the robots are guided by the rows of crops. Slaughtershows a robot with two guidance cameras attached three meters apart toprovide robust performance in widely spaced row crops, such as tomatoes.

It is an object of this invention to simplify the identification ofweeds and simplifying the autonomous robot for application to anaesthetic mulch garden. It is a further object of this invention toprovide a low-cost autonomous robotic weed removal system which isappropriate for the residential and commercial markets by downsizing therobot and using simpler and more economical technology.

Automating with robotic systems tasks that were hitherto laboriouslyperformed by humans is often desirable because it frees humans to domore rewarding and productive work. In many cases, robots can performtasks faster, more accurately, and with better efficiency. Gardenmaintenance can incur very high labor costs and the use of robots canoften reduce the expense of garden maintenance substantially. Robots canperform tasks safely that can be hazardous for humans such as removingweeds from snake or wasp infested mulch beds.

The use of herbicides to remove weeds in aesthetic mulch gardens candestroy the ecologic balance and sicken wildlife. Herbicides can harmanimals such as birds, squirrels, lizards, chipmunks, and rabbits.Additionally, they harm the overarching environment through infiltratingground water. Personnel applying herbicides are subject to breathing andingesting carcinogens which can be detrimental to their health. The costof herbicides can be expensive, and the porosity of common mulch can behigh and require large volumes of costly chemicals. It is therefore anobject of the present invention to eliminate the use of herbicides andpoisons to kill weeds.

As described by Koselka et al (U.S. Pat. No. 7,854,108), agriculture isone industry with traditionally low profit margins and high manual laborcosts. In particular, harvesting can be expensive. For some crops, suchas tree fruit, harvesting labor represents the growers' single largestexpense, up to 50% of total crop cost. Increasing labor costs and laborshort ages threaten the economic viability of many farms. Therefore,replacing manual labor with robots would be extremely beneficial forharvesting. Koselka '108 further stated that GPS controlled automatedtractors and combines already operate in wheat and other grain fields.Automated harvesters exist that can blindly harvest fruit by causing thefruit to drop from a plant into a collection device. There is anabundance of prior art directed to automated large-scale harvesting ofcrops in farms. Such automated systems usually comprise a GPS sensor orother external navigational aids to determine position. Machine visionsystems are known in the prior art. One known approach for automatedharvesting of fresh fruits and vegetables and the like is to use a robotcomprising a machine-vision system containing cameras. The cameras maybe utilized to identify and locate the fruit on each tree and weedsaround plants. The vision system may be coupled with a picking system orother task specific actuators to capture data from various locations inand around each plant when performing the picking or desiredagricultural function. Calleija et al (U.S. Pat. No. 10,701,852)describes an automatic target recognition and management system forremoving weeds using a tine that is actuated to swing in an arcuatedirection to disrupt a targeted weed. Calleija '852 teaches that it iswell known that one of the most important aspects in agriculture andcrop cultivation is the effective management of weeds. He states thatweeds compete aggressively for limited resources in terms of space,water, sunlight, and nutrients. Consequently, the emergence of weeds hasa marked adverse impact on crop yield and quality. Moreover, due totheir relatively fast growth rates compared to crops, if weeds are noteliminated or effectively managed, particularly during the preliminarystages after crop planting, they can quickly dominate entire fields andresult in serious yield losses. Calleija '852, as in much of the priorart, is focused on large-scale farming. Calleija '852 further teaches adevice for weed removal which has a multipronged tine which moves aboutat least one control axis in a predetermined direction between anengaged position wherein the tine formation in use contacts a groundsurface for removal or destruction of targeted weeds, and a disengagedposition wherein the tine formation is substantially retracted from theground surface ['852: col. 6, ln 13-19]. A further aspect of Calleija'852 is a vacuum system having a suction device (e.g., nozzle) arrangedadjacent to the weeding implement ['852: col. 6, ln 37-38]. Calleijateaches a very sophisticated machine vision and analysis system foridentifying and mapping a variety of plants and weeds and selectivelytargeting weeds for removal, while leaving the crops undisturbed.

Yuan Biao et al. (CN111109237A) discloses an automated robot system foragricultural weeding based on camera recognition utilizing a comparisonmodule which compares weed images with crop images and therebyidentifying weeds to be removed while crops are prevented from beingremoved by mistake. Yuan Biao '237 discloses weed removal by means ofrobots having chemical agents to spray on weeds. A light source moduleis used for supplementing light during target identification. Theweeding method seeks to avoid damaging crops as weeds are sprayed withherbicides and chemical agents. The robot system is based on a mobileterminal which interfaces with a stationary base station having acentral processing unit to direct operations. The stationary basestation and the mobile weeding robot communicate with a bidirectionalsignal connection.

It is important to note that agricultural weed removal solutions usingthe aforementioned large intra-row agricultural robots employ veryadvanced technology incurring substantial equipment and operating costswhich, while cost-effective in large agricultural operations for thereasons previously discussed, are far too expensive for robotic weedremoval in aesthetic mulch gardens. The robotic weed removal system foraesthetic mulch gardens described in this specification provides acost-effective solution with relative simplicity which can provideeffective weed control in aesthetic mulch gardens found in gardenstypically found in residential, retail, and commercial environments.

Aesthetic gardening is extremely common in urban and suburban areas andis designed to beautify a home or commercial enterprise. Aestheticgardens have flowers, shrubs, rocks, wooden structures such as railroadties or tree stumps, ceramic statues, and the like. The garden isusually bounded by a large grassy area or by a brick or stone patio.Rather than have the soil directly facing the surrounding atmosphere, alayer of mulch helps protect the soil from baking in direct sunlight andit helps the soil hold moisture so that their watering requirements arereduced. Most sources recommend a layer of from 2-3 inches of mulch overthe soil. FIGS. 3 and 13 show exemplar mulch gardens.

Mulch is any material that covers the soil's surface. It can includeshredded or chipped bark, straw, or hay, fallen leaves and plant debris,compost, wood chips, rotted manure, cardboard, or even seaweed.Depending on the type of mulch, it can be used to beautify the gardenwith rich dark contrasting color between the ground and flowers andshrubs. Mulch is usually dark and of uniform color. Maintainingaesthetic mulch gardens is a tedious process because of the continualemergence of small, localized weeds which can overwhelm a garden ifweeds are not removed frequently as they emerge. When done manually,weed removal is a tedious time-consuming process with high labor costs.Weed removal in mulch gardens generally requires a person to crawlaround on his/her hands and knees to remove, one-by-one, each weed.Also, mulch gardens can be home to various creatures including wasps andsnakes, which can make hand-weeding mulch gardens hazardous.

There is a dearth of literature teaching the use of autonomous roboticweed removal to address this important issue. It is an object of thepresent invention to address this need by producing technology whichallows frequent removal of weeds as they form and while they are smallso as to eliminate the need for humans to remove the weeds by hand andto maintain the aesthetic mulch garden in a nearly pristine weed-freecondition.

It should be noted that by covering the soil, mulch naturally reducesweed growth, which is a major source of its popularity in domestic andcommercial applications. However, as previously stated, slowly butsurely, weeds do grow in mulch beds and must be maintained or else themulch bed becomes unsightly with an abundance of weeds. If the mulch bedis frequently maintained, unsightly weed presence can be minimal and theweeds that do emerge can be extracted while they are relatively small,and their roots have not penetrated deeply into the soil. However, if itis not regularly maintained, weeds can become abundant and large anddestroy the aesthetic value of the mulch bed and become harder to removeas their root systems become deeper and more mature.

If the mulch bed is maintained at frequent intervals such that weedsremain relatively small, preferably with a footprint smaller than 1-2inches, the forces required to remove weeds from mulch beds are small.It is an object of the current invention to utilize the low forcerequirements in extracting emergent weeds which is inherent to mulchgardens to simplify and reduce the cost of robotic extraction andmaintain the size of the robot fairly small so as reduce cost andimprove its ability to follow trajectories between typical obstaclesfound in mulch gardens. Said force requirement is directly related tothe size of the weed since it is desirable to effectively remove theentire root system of the weed during extraction, and this requireddeeper penetration of the soil, which requires more force. Large forcesrequire heavier robots to counter the consequentially larger reactionforces of weed extraction devices that must penetrate deeper into thesoil to extract the root system of the weeds. Hence, it is an objectiveof this invention to provide a relatively small, light-weight robot,which can remove small weeds along with their root systems efficientlyand with sufficient frequency such that the weeds are not allowed togrow to large sizes.

While weed removal from large-scale agricultural facilities and weedremoval from relatively small aesthetic mulch gardens have manysimilarities, the above-described characteristics show that the twoapplications are different in many important respects and that weedremoval technology in an aesthetic mulch garden lends itself to noveltechnologies that are not viable in the large agricultural setting, andvice versa.

The typical characteristics of an aesthetic mulch garden are (1) theyhave generally small areas in comparison to the lawn or patio that theytypically are adjacent to; (2) the mulch used in aesthetic gardens isusually of fairly dark and uniform color. Mulch can often be purchaseddyed and in a chosen color; (3) mulch is often laid on fairly levelground since attempting to lay it on steep hills with large inclinationsinvites runoff when it rains; (4) flowers and shrubs planted in mulchgenerally protrude substantially above the mulch bed and are easilyvisible. Unlike the agricultural application where discriminatingbetween a weed and a crop may be very difficult and requirestate-of-the-art machine vision techniques, in an aesthetic mulchgarden, weeds are very much smaller than shrubs or flowers, and caneasily be discriminated on the basis of size, provided there is frequentmaintenance and the weeds are not permitted to grow large; (5) mulchbeds are often delineated with some border material such as plastic orbrick; (6) Weeds that appear in mulch beds are usually highly visiblebecause of the contrasting color between the weed and the mulch, whichis a reason for considering them to be unsightly; (7) the morefrequently weeds are removed from the mulch bed, the smaller the weeds,and the lower the force requirements to pull them out, and the smallerthe volume of the removed mulch, and the smaller the robot. All of thesefeatures combine to provide a specialized robot meeting the objectivesof this invention.

It is an object of the present invention to simplify the technologydeveloped for autonomous robotic weed removal in agriculture to adapt tothe needs of an entirely different market segment: the residential andcommercial aesthetic mulch garden maintenance. A further object of thepresent invention is to simplify the task of machine vision todiscriminate between a small, usually green, weed and a dark, usuallybrown or black, mulch, rather than to discriminate on more sophisticatedclassification and mapping. A further object of the present invention isto disclose a small, lightweight, autonomous robotic weed removal systemthat takes advantage of the unique characteristics of an aesthetic mulchgarden as described above. A further objective of this invention is toprovide an effective robotic weed-search trajectory appropriate for anaesthetic mulch garden based on reflective randomization strategy ratherthan the intra-row guidance strategy used in agricultural robotic weedremoval. A further object of the present invention is to simplify themachine vision requirements by exploiting the natural visible colorcontrast between weeds and mulch. A further objective of the presentinvention is to use low-cost modern sensing technology coupled with asmall light-weight robot to provide an economical solution tomaintaining aesthetic mulch gardens. A further objective of the presentinvention is to help the environment by using mechanical weed removaland avoiding herbicides and chemicals.

It should be noted that weed removal methodologies rarely providecomplete extraction of a weed, despite the desirability of doing so.Generally, a small fraction of the weed root system may remain in thesoil and continue to grow as soon as the robot has extracted a largeportion of the weed. The complete removal of a weed during an extractionprocess, while desirable, is not necessary, nor is it probable. A personof ordinary skill in the art would understand that this is yet anotherreason why it is important that robotic weed extraction be a frequentlyrepeated process. The rapidity of this reemergence of weeds depends muchon the type of weed, the depth of its root system, the amount ofirrigation provided, the nature of the mulch, the amount of sunlight,and the effectiveness of previous weed removals, and other factors.These factors, in-turn, determine the frequency to which this inventionmust be utilized.

SUMMARY OF THE INVENTION

A system and method for removing weeds from an aesthetic mulch gardenwith a domain having predefined boundaries which comprises a wirelessautonomous robot controlled by a central processing unit (CPU) andpowered by an on-board electrical power supply such as a battery.Controlled by the CPU, the robot has a navigation module, a propulsionmodule, and a weed extraction module.

The navigation module has three levels of sensor which inform the CPU:(1) A domain boundary detection system which confines said robot to saiddomain, and whereby said robot is programmed to follow a substantiallylinear trajectory. Sensors continuously seek the boundaries of saiddomain; (2) An independent collision avoidance system with a sensorwhich detects trees, shrubs, statues, landscaping rocks, lighting, andother objects present in an aesthetic mulch garden which might interferewith the robot's search protocol; (3) A machine vision system with acamera which identifies weeds and positions the robot accurately toenable removal or destruction of said weeds.

The said domain boundary detection system, in order to restrain thetrajectory of said robot to the interior of said domain, the inventionincludes a means for defining the boundaries of said domain. The userdefines said boundaries prior to activation of said robot and prior tothe inception of the robot's weed removal process. In accordance withthe invention, the user can define the domain of the aesthetic mulchgarden by a variety of methods which includes: defining the boundary ofsaid domain by implanting at regular intervals magnetic stakes on theboundary of said domain and equipping said robot with a magnetometer todetect the magnetic field generated by such magnetic stakes; definingthe boundary of said domain by burying a conductor which transmits radiowaves around the boundary of said domain and equipping said robot with aradio receiver to detect such radio signals; programming the positioncoordinates of the boundary of the domain into said CPU controlling saidrobot which can be positioned with a control system using GPStechnology; surrounding said domain boundary with LED rope lights whichare illuminated by an independent power supply and equipping said robotwith a photodetector to detect such luminous signals from the LED rope;and other methods which define said boundary in terms that can beprocessed by the CPU in conjunction with a suitable sensor to determinewhen the robot is approaching the boundary. In the event that a boundaryis detected by the domain boundary detection system, CPU commands saidrobot to stop and turn by a prescribed angle, followed by an instructionto continue on a linear trajectory at a prescribed speed.

In accordance with this invention, the overall trajectory of said robotmust be such that it searches the domain in its entirety. In accordancewith this invention, it achieves this by means of following a reflectiverandomized trajectory. Upon activation, said robot commences itstrajectory linearly. It continues linearly until a boundary, anobstacle, or a weed is detected by a sensor. At that point, the CPUcommands the robot to stop. If the robot encounters a weed, the protocolof the weed machine vision system is invoked. If the weed is removed, orthe robot encounters a boundary or obstacle, the CPU orders said robotto pivot on its vertical axis by the reflection angle (a fixed anglespecified in the navigation module). Said robot the resumes a lineartrajectory until either a weed, a boundary, or an obstacle isencountered whereby the process is initiated repeatedly until the entiredomain has been searched, a timer has reached a preprogrammed timelimit, or the user deactivates said robot. Thus, the current inventionremoves weeds throughout the domain by means of a search strategy ofrandomly reflecting from detected boundaries and obstacles through alarge plurality of encounters until a large fraction of the domain issearched.

A collision avoidance system is included with an independent sensorwhich mounted in the forward-facing portion of said robot to detectlarge obstacles relative to the weed size which might interfere with therobot's search protocol or potentially damage the robot. The field ofview of the sensor is equal to or larger than the width of said robotand has a depth of field no smaller than the stopping distance of therobot. It is not necessary for the collision avoidance system to providean accurate image of the obstacle but is intended to enable said robotto detect trees, shrubs, statues, landscaping rocks, lighting,structures, and other objects that may be present in an aesthetic mulchgarden and issue warning signals to the CPU. In the event that such anobject is detected, CPU commands said robot to stop and turn by aprescribed reflection angle, followed by an instruction to continue on alinear trajectory.

The weed machine vision enables the robot to detect weeds within saiddomain and to position said robot for weed removal. The weed machinevision system includes a camera which is directed at an angle towardsthe ground where it is focused. The field of view of the camera is atleast twice as large as the anticipated size of the weed. Since themulch is expected to be of a dark nonhomogeneous color and the weedsnormally have a substantially different contrasting color with respectto said mulch, color discriminating software within the CPU can identifythe presence of weed based both on color and size. To enhance thecolorimetric contrast between mulch and weed, and to enable theinvention to be used in the dark, a lamp which illuminates the area ofview of the machine vision camera may be added and fixed to said robot.To further enhance contrast, optical filters may be used on the saidlamp. Since the invention is contemplated to be used frequently forgarden maintenance, the system is intended to detect weeds at theirinception when they begin to emerge from the surface of the mulch whenthe weeds are relatively small. Therefore, the size of the weed isanticipated to be smaller than said field of view of said camera.Similarly, shrubs, which are much larger than the field of view of saidcamera, can be identified by the discriminating software in the CPU tonot be identified as a weed based on its size. Once a weed isidentified, the CPU with input from the machine vision camera accuratelypositions said robot for weed removal or destruction.

The propulsion module has a motor controller that can receive commandsfrom the CPU to control said robot drive motors so as to enable saidrobot to translate, turn, start, and stop, adjust speed, as well asother propulsion commands as may be needed. Said robot preferably mayhave three wheels or four wheels. The robot may also use alternativemethods of direct propulsion such as artificial legs, propellers, andcrawler wheels as seen on cranes and military tanks. In oneconfiguration, the rear wheels drive the robot and are powered by anelectric drive motor, while the front wheels provide steering, and thesteering mechanism is controlled by a servo motor. Alternately, thedriving motor can be in front of the robot, while the steering servomotor can be at the rear. Both drive motors and steering servo motorsare controlled by a motor controller which receives instructions fromthe CPU and receives power from said on-board electric power supply.Another configuration which provides better, and simpler control is tohave the two front wheels of said robot being driven by two independentelectric drive motors while both rear wheels are driven by oneindependent drive motor. All three drive motors are controlled by themotor controller which receives commands from the CPU. With this latterembodiment, the need for a mechanical steering mechanism is eliminatedand turning can easily be accomplished by directing each front wheel torotate at different speeds. To achieve very precise turning, one drivewheel can be directed to rotate in one direction while the other frontdrive wheel is directed to rotate in the opposite direction. One ofordinary skill in the art can easily find other combinations of drivewheels and drive motors which fall within the teaching of thisinvention.

The weed extraction module causes either complete extraction ortermination of the growth of the weed in the aesthetic mulch garden. Aspreviously discussed, said robot searches for weeds as described so thatthe CPU directs the robot to position itself directly above the weed atthe proper location for extraction or termination. In one configurationthe weed removal is done with a servo-controlled 2-axis multi-clawedgrabber to grip weeds and extract them from the mulch bed. Othermechanical methods where a weed extraction device is mounted on saidrobot and is directed by the CPU to cause extraction are within thescope of this invention. As described by Calleija et al. (U.S. Pat. No.10,701,853), such methods may include end effectors mounted to saidrobot which take the form of a tine, sometimes arranged in plurality, anauger, a hoe, a scythe, a knife, a cultivator, a fork, a brush, a diskor other implement which may scoop or rotate in a manner in which a weedis removed from the ground. In addition, the literature shows nonmechanical methods of weed extraction and termination (e.g., Slaughteret al.) including delivering herbicides broadly with sprays, or locallywith jets and microjets, energetic laser beams, electric arcs, etc. Thepresent invention may be adapted to include these and other means ofweed extraction or destruction since they can be attached to an endeffector in said robot. The present invention includes a built-insuction system to remove the weed from mechanical weed removal apparatusand deposit it in a receptacle mounted on said robot for later disposal.Said suction system can be controlled by the CPU to operate for a shortperiod of time after the weed is removed which further cleans themechanical extraction device and readies it for its next deployment. Therobotic propulsion, navigation, collision avoidance, weedidentification, weed removal, and other actions are autonomouslycontrolled with one or more digital central processing units whichreceive input from all sensors.

The robot for use in weed removal in an aesthetic mulch garden must besmall enough so that it can pass through and around a multiplicity ofobjects including shrubs, statues, trees, structural elements, and thelike which expected to be present due to the artistic nature of mulchgardens. Said robot must be capable of deftly maneuvering in a tightenvironment where the domain is highly irregular in shape and there maybe many obstacles. Said robot must be stable as it traverses an unevengroundcover of mulch which may be soft and spongy, and said robot mustbe capable of positioning itself fairly accurately. Said robot must besufficiently robust to carry several motors, servos, a battery or otherelectrical power supply, a CPU with various electronics, steeringmechanisms, a receptacle for removed weeds, a vacuum device forproviding suction, and to have the heavy-duty structural elements neededto support all of this equipment. The weight of said robot must besufficient to dominate the vertical reaction forces generated by theweed extraction module, which employs a specific extraction method, sothat it may function properly and effectively remove weeds includingtheir roots from an aesthetic mulch garden. Further, it will preferablybe equipped with protective covering for the components or housing. Saidcovering may be resistant to weather since it is contemplated that theremoval process may take place outside during rain or other inclementweather.

In operation, upon being activated by the user, said robot will patrolthe defined domain of the aesthetic mulch garden, following a randomreflective trajectory, whereby the robot reflects at a prescribedreflection angle from boundaries and obstacles leading to a randomizedtrajectory. During the course of following this trajectory, said robotsearches for weeds by employing machine vision based on color contrastbetween weeds and mulch, and it removes weeds when encountered whileavoiding shrubs, statues, structures, rocks, and other obstacles, andconfining its operations to the interior of a domain that is defined bythe user.

The present invention is in distinct contrast to prior art whichdisclose methods for weed removal in large-scale agriculturalapplications where crops are planted in regular, substantially linearrows. The agricultural robot trajectory generally follows substantiallystraight rows of crops and prior art agricultural systems require highlysophisticated mapping systems to discriminate between weeds and crops,both of which may have similar coloring and size.

BRIEF DESCRIPTION OF THE FIGURES

This patent and application file contains at least one figure executedin color. Copies of this patent or patent application publication withcolor figures(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a PRIOR ART agricultural autonomous intra-row weed controlsystem [Slaughter et al 2007].

FIG. 2 is a schematic of a PRIOR ART trajectory of an agricultural weedcontrol robot by intra-row guidance.

FIG. 3 is a schematic of the trajectory of a mulch garden weed controlrobot by reflective randomization in accordance with the preferredembodiment.

FIG. 4 is a side view of a mulch garden weed control robot in accordancewith the preferred embodiment.

FIG. 5 is a top orthographic view of the mulch garden weed control robotshown in FIG. 4 .

FIG. 6 is a bottom orthographic view of the mulch garden weed controlrobot shown in FIG. 4 .

FIG. 7 is a schematic of the control system architecture for the mulchgarden weed control robot shown in the embodiment of FIG. 4 .

FIG. 8 is a process diagram showing the control of navigation, search,and weed removal for the mulch garden weed control robot shown in theembodiment of FIG. 4 .

FIG. 9 shows the steps of the weed extraction process for the mulchgarden weed control robot shown in the embodiment of FIG. 4 .

FIG. 10 shows the dual servo operation controlling the “grabber claw”weed extraction device for the mulch garden weed control robot shown inFIGS. 4 and 9 .

FIG. 11 shows the results of a sample calculation to determine thepreferred spacing for the magnetic stakes using a representative type ofpermanent magnet.

FIG. 12 shows the relationship between the machine vision camera fieldof view and the longitudinal axis of the robot.

FIG. 13 Shows a photo of a typical aesthetic mulch garden indicatingweeds to be extracted and various objects such as shrubs and trees thatmust be distinguished and bypassed by the robot.

FIG. 14 Shows a sequence whereby in FIG. 14(a), the machine visioncamera can photograph a weed at high resolution (300 pixel/in), and FIG.14(b) shows the same photograph at low resolution (5 pixel/in), and FIG.14(c) shows the same pixels for the green weed, but background pixelscorresponding to the mulch are converted to black.

FIG. 15(a) Shows a low-resolution field of view containing a weed,corresponding to FIG. 14(c), having a blackened background and the Fieldof View coordinate system employed to identify the location of thecentroid of the weed.

FIG. 15(b) Shows the relationship between the robot weed grabber clawpositioning coordinates and the camera Field of View coordinates.

FIG. 16 Shows the process by which the CPU in coordination with themachine vision identifies a weed.

DETAILED DESCRIPTION OF THE INVENTION

As seen in FIG. 3 , this invention discloses a system and method forsearching for and removing weeds 3 from an aesthetic mulch garden 5having a domain 10 with a closed user-predefined boundary 12 whichcomprises a wireless autonomous robot 15 controlled by a centralprocessing unit (CPU) 100 and powered by an on-board electrical powersupply (not shown) such as a battery. A preferred embodiment of saidrobot 15 is shown in detail in FIGS. 4, 5, and 6 . FIG. 7 shows thecontrol system with said CPU 100 receiving inputs from domain boundarydetector 125, collision avoidance detector 40, and weed detector 35 andsending outputs to the propulsion module 120, and the weed extractionmodule 130. Robot 15 has a base 95 which forms the central structuralelement to which all hardware associated the robot navigation module110, the weed extraction module 130, the propulsion module 120, theelectric power supply 115 and the CPU and related sensors, cameras, andelectronics are mounted.

In the preferred embodiment, the CPU 100 is a Raspberry Pi 4 Model B,which has a Broadcom BCM2711 quad-core Cortex-A72 which is a 64-bitSystem on a Chip (SoC) which operates at 1.5 GHz. This CPU has with 2 GBSDRAM and Bluetooth enabled. It includes a Videocore VI GraphicsProcessing Unit (GPU) which enables graphical input/output such as inputfrom machine vision. It also has multiple USB3.0 inputs/outputs forsimultaneous operation of multiple peripheral devices including sensors(e.g., 35, 40, and 125), motor controllers (e.g., 82), and servos (e.g.,300, 305). The Broadcom unit is equipped with a Wi-Fi antenna whichenables direct commands from a user with a Bluetooth transmitter (e.g.,cell phone, tablet) to the CPU 100 to control said robot 15 while inoperation if desired. The Raspberry Pi 4 CPU is designed to beconfigured with a machine vision camera 35 such as the SONY IMX219PQH5-C8 Mega-Pixel CMOS Image Sensor (herein referred to as a machine visioncamera) with square pixels. The color system of the said SONYIMX219PQH5-C camera utilizes the R, G, and B primary color pigmentmosaic filters and has an electronic shutter with a variable speed. TheRaspberry Pi 4 can be programmed in C/C++, Python 2/3, and Scratch bydefault. However, nearly any language compiler or interpreter can beinstalled on the Raspbian Operating System which controls the RaspberryPi 4.

As further seen in FIG. 3 , in the preferred embodiment, the domain 10of the aesthetic mulch garden 5 is defined by the user by placing aplurality of magnetic stakes 20 on the boundary 12 of said domain 10with a spacing 22 that is roughly regular. As shown in FIG. 7 , when theautonomous robot 15, following its trajectory 25 and under the scrutinyof the robot navigation module 110, comes into proximity to one of themagnetic stakes 20, the domain boundary detector 125, which in thepreferred embodiment is a digital positioning multi-axis magnetometer,detects one or more magnetized stakes 20, which then signals the CPU100. As shown in FIG. 8 , the CPU 100 then commands S215 said robot 15to stop and reflectively turn by a prescribed reflection angle 27, inthe range 60°-120°. In accordance with S205, CPU 100 then commands saidrobot 15 to follow a linear trajectory 25 in the reflected direction. Asseen in FIG. 3 , the CPU thereby directs the movement of said robot 15to patrol for weeds 3 with a trajectory 25 constrained inside saidboundary 12 and continues to repeatedly cross the domain 10 following aseries of random reflections until either a preprogrammed timelimitation is met, a minimum frequency of weed detection criteria ismet, or the robot 15 is deactivated by the user or by other criteriawhich may be programmed into CPU 100.

In alternative embodiments, the domain boundary detection system 125 canuse other methods by which the user can define the boundary 12 of thedomain 10, and, corresponding to those methods, an appropriate domainboundary detector 125 can be used. Examples of such domain boundarydetectors 125 and methods include: (1) burying an electrical conductorwhich transmits radio waves around the boundary 12 of said domain 10 andhaving a domain boundary detector 125 mounted in said robot 15 be aradio receiver to detect such radio signals, and when said robot 15 isin close proximity to said boundary 12 as shown in FIG. 8 , saiddetector 125 then signals CPU 100 which issues commands in accordancewith FIGS. 7 and 8 ; or, (2) having the user program said CPU 100 withthe position coordinates of the boundary 12 of the domain 10 andcontrolling said robot 15 on trajectories inside the boundary 12 usingin a domain boundary detector 125 GPS technology or other wirelessposition coordinate measurement technologies to determine when saidrobot 15 is in close proximity to said boundary 12 as shown in FIG. 8 ,said detector 125 then signals CPU 100 which issues commands inaccordance with FIGS. 7 and 8 ; or, (3) user placing on said boundary 12LED rope lights which are illuminated by an independent power supply andhaving a domain boundary detector 125 mounted in said robot 15 be aphotodetector to detect such luminous or chromatic signals from the LEDrope detect such luminosity, and when said robot 15 is in closeproximity to said boundary 12, as shown in FIG. 8 , said detector 125then signals CPU 100 which issues commands in accordance with FIGS. 7and 8 . The scope of the invention includes other similar methods wherethe user can define said boundary 12 which can be sensed bycorresponding boundary detection systems 125 which can provide input tosaid CPU 100 which controls said robot 15 as taught in this invention.

In the preferred embodiment, the robot 15 searches for weeds 3 with thenovel randomized reflective trajectory scheme described above, wheresaid trajectory 25 is shown in FIG. 3 whereby the linear trajectory 25of robot 15 reflects from the boundary 12 at a fixed reflection angle 27when encountering the magnetic signal from the magnetic stakes 20 at theboundary 12. A reflection angle 27 of 100 degrees is used, although theuser may wish to program the CPU with different values depending on thesize of the domain 10, the level of weed 3 removal (generally less than100%), and the time desired to complete the weed removal process. Themagnetic stakes 20 placed on the boundary 12 of said domain 10 havepermanent magnets attached above the ground, preferably not more thanone inch for aesthetic reasons. The magnetic field strength of themagnets, the spacing 22 between said magnetic stakes 20, and thesensitivity of the magnetic field strength that said magnetometer 125 iscapable of measuring may be varied to obtain a clear signal when saidrobot 15 approaches said boundary 12. The magnetic flux density, B, maybe calculated for a specified permanent magnet, magnet stake spacing 22,and distance from the boundary. Design should be based on the weakestsignal which occurs at the midpoint between magnetic stakes 20. Thefollowing equation determines the magnetic flux density, B, along a lineperpendicular to the boundary 12 midway between said magnetic stakes 20,in the horizontal plane:

$\begin{matrix}{B = \frac{D^{2}LB_{r}}{{8\left\lbrack {x^{2} + \left( \frac{d}{2} \right)^{2}} \right\rbrack}^{3/2}}} & \left( {{Equation}1} \right)\end{matrix}$

where:

-   -   B=magnetic flux density (Gauss)    -   B_(r)=residual magnetic flux density (Gauss)    -   D=diameter of disc type permanent magnet    -   L=thickness of disc type permanent magnet    -   d=spacing 22 between magnetic stakes 20    -   x=distance from boundary 12 measured perpendicular to midpoint        between magnetic stakes 20

A sample calculation using Equation I is shown in FIG. 11 for acommercially available neodymium permanent magnet in the form of a discof 0.75 inch diameter and 0.375 inch thickness with axial polarity wherethe magnetic flux density, B, was calculated as a function of theperpendicular distance, d, from said boundary 12 in the horizontalplane, for three representative spacings 22 of said magnetic stakes 20.Note that the residual magnetic flux density, B_(r), is a physicalproperty of the magnetic material and is generally available as aspecification for a given permanent magnet. From the calculation shown,the preferred choice of spacing for said magnetic stakes 20 is 9 inchesbecause the said robot 15 should be commanded to stop at about 2 inchesfrom the boundary. With a 9-inch spacing, the magnetic flux density isabout 3.0 Gauss which is well within the operating range of the +/−8.0Gauss specification on the selected magnetometer within said domainboundary detector 120. When domain boundary detector 120 signals CPU 100that a magnetic field of say 3.0 gauss is detected, CPU 100 would thencommand robot 15 to stop as in S210 shown in FIG. 8 . CPU 100 would thenproceed issue a command to stop and turn said robot 15 the prescribedangle of 100° according to S215.

One of ordinary skill in the art would recognize that the optimumspacing would depend on type of material from which the magnet is made,the residual magnetic flux density, B_(r), the shape of the magnet, thediameter and thickness of the magnet, as well as the sensitivity of themagnetometer used in said domain boundary detector 125. The cost of themagnets would also be another factor in deciding on the optimal spacingas the cost increases with the number of magnets, the size of themagnets, and the material of the magnets.

In the preferred embodiment, the results of the calculation of FIG. 11are used. Said magnetic stakes 20 are spaced at about 9-inch intervals,and the magnets mounted on said magnetic stakes 20 are neodymium and ofcylindrical in shape with 0.75-inch diameter and 0.375-inch thicknesswith axial magnetization. The polarization axis of said magnet is axialand attached to the top of the stakes with the axis vertical. Whileneodymium magnets were selected in the preferred embodiment, because oftheir strong magnetic field, low cost, permanency, and robustness, manyother types of permanent magnet are within the scope of this invention.These include ceramic ferrites alnico, rare earth magnets, and others.In other embodiments, magnetic stakes 20 could also includeelectromagnets. Since the magnetic field strength decreases as the cubeof the distance from the magnet, and the magnetometer must detect themagnetic field as it approaches said boundary 12, the magnetometer thatis used must be capable of detecting a much lower level of magneticfield than that surrounding said magnetic stake 20. Thus, in thepreferred embodiment, a ±8 gauss 3-axis magnetometer such as the MEMSICMMC5883MA is used which can detect the magnetic field at least threeinches from said boundary 12. To most accurately define the contours ofsaid boundary 12, while minimizing the number of magnetic stakes 20needed, there is a tradeoff. Clearly, the smaller the spacing 22 betweenmagnetic stakes 20, the better the definition of the boundary 12.However, in the aesthetic garden application, defining the boundary 12with great precision may not be needed, while minimizing the number ofmagnetic stakes for aesthetic and cost reasons may be a dominantconsideration. The smaller the spacing of the magnetic stakes 20, thegreater the number of magnetic stakes 20 required, which adds to thematerial cost, the labor cost of installation, and the generalappearance of the boundary 12 of the aesthetic mulch garden 5.

As seen in FIGS. 3 and 13 , an aesthetic mulch garden 5 may have manyobjects which may obstruct said robot 15 as it patrols the domain 10 insearch of weeds 3. These include trees, shrubs, rocks, statuary,structural elements, and the like. In order to enable said robot 15 toavoid interaction with these objects, the robot navigation module 110includes a collision avoidance detector 40, as seen in FIGS. 4, 5, 7 .As seen in FIGS. 7& 8 , when said collision avoidance detector 40locates an object larger than a preprogrammed dimension, it sends asignal to said CPU 100, which then commands said robot 15 to stop, turnby the said preprogrammed reflection angle 27, which is 100 degrees inthe preferred embodiment, and continue on a linear trajectory 25. In thepreferred embodiment, said collision avoidance detector 40 is anultrasonic distance measuring range sensor. A commercially availablecollision avoidance sensor 40 which may be used in the preferredembodiment is the TDK CH101 Ultrasonic Range Sensor. This embodiment ofcollision avoidance sensor 40 has a customizable field of view which canbe made narrow enough to limit the view to objects on the trajectory ofsaid robot 15. Said collision avoidance sensor 40 can detect objectswithin a range of 4 cm to 1.2 m and has programmable modes optimized formedium and short-range sensing applications. Said embodiment ofcollision avoidance sensor 40 works in any lighting condition, includingfull sunlight and complete darkness, and it is insensitive to objectcolor. Said embodiment of collision avoidance sensor 40 is fullyprogrammable to detect objects in a larger size range so as to belimited to the larger objects such as shrubs, rocks, trees, and thelike, and not be sensitive to smaller objects such as weeds. In otherembodiments of this invention, commercially available radar rangesensors or laser range sensors may also be used as collision avoidancesensors 40 and are well within the scope of the present invention.

As best seen in FIG. 6 , the preferred embodiment of said robot 15 isalso equipped with machine-vision camera 35 that identifies and locatesweeds based on chromatic contrast in visible radiation emanating fromthe mulch and the weeds. As best seen in FIG. 12 , said machine-visioncamera 35 is focused on the ground a short distance ahead of said robot15 with a controlled camera field of view 370 which can be adjusted bythe user with a suitable lens hood extension (not shown) attached to thelens of said machine-vision camera 35. A lamp (not shown), which can beactivated and deactivated by CPU 100 as programmed by user, may bedirected to illuminate said field of view 370 so as to enhance contrastbetween weeds and mulch, and to enable operation of said machine-visionweed detector 35 in the dark. Such lighting may comprise light emittingdiodes or incandescent bulbs and include optics which focus saidillumination on the camera field of view 370 and may also includepolarizers and/or filters acting on the illumination so as to enhancecontrast. As seen in FIGS. 7, 8, and 9 , when said weed machine-visiondetector 42 detects a weed 3 as indicated in S220, a signal is sent toCPU 100 which then commands said robot to position itself S225 and S230for weed 3 removal. CPU 100 then activates and commands the weedextraction module 130 in accordance with steps S235, S240 for weed 3removal. The following paragraph explains how this might be done in onepreferred embodiment.

In the preferred embodiment for locating weeds 3 in a mulch garden 5,the camera field of view 370 has dimensions of 6 inches in width and 4inches in depth as is schematically shown in FIG. 12 . The center ofsaid camera field of view 370 is located about 12 inches directly infront of said robot 15. The user might alter these dimensions inaccordance with the specific application and in accordance with thespeed vs. accuracy tradeoffs that a person of ordinary skill in the artmight deem necessary.

In the preferred embodiment, the CPU 100 is a Raspberry Pi 4 Model B,with a machine vision camera 35 such as the SONY IMX219PQH5-C 8Mega-Pixel CMOS Image Sensor with square pixels which utilizes the Red,Green, and Blue (RGB) primary color pigment mosaic filters as describedabove. RGB is a widely known system in which the primary colors, Red,Green, and Blue can produce a vast array of colors by additivesynthesis. Colors are coded digitally as an RGB file where an arbitrarycolor can be stored digitally in three bytes of data, each having 8 bitsor 256 possible values. Thus, an 8 bit per channel RGB file has one bytefor red which is R=0 to R=255, a second byte for green is G=0 to G=255,and a third byte for blue is B=0 to B=255. Blending the colors, an RGBfile for black would be R=0, G=0, and B=0. An RGB file for white wouldbe R=255, G=255, and B=255. In general, and RGB file can be written as atriplet: (R, G, B). Thus, a brownish color can be coded as (171, 122,43). Given all of the possible values for R, G, and B, this systemprovides 256×256×256=16,777,216 possible colors. It should also be notedthat while the 8 bits per channel RGB system is most common, othersystems providing more colors include 12-bit, 16-bit, 24-bit, and 32-bitRGB systems. Also, there are many other systems for digitizing colorssuch as the CMY or CMYK color models. In the preferred embodiment, an8-bit per channel RGB system is used but the invention is conceived toinclude all other such systems which are well known to those skilled inthe art.

In the preferred embodiment, the machine vision weed detector 42utilizes the fact that mulch gardens 5 are generally dark in color,while weeds 3 are generally green in color and small in size (assumingthat the mulch garden is frequently maintained so as to avoid excessiveweed growth). Thus, by considering the weed color, dark mulch color, thecontrast of weed 3 with the mulch, and the size of the weed, a weed canbe discriminated. This is a very simple means to locate weeds as opposedto the complex plant identification systems and mapping that waspreviously discussed in Slaughter et al. (Elsevier, 2007), whichrequires a very sophisticated and expensive sensing system to measureimportant physical and biological properties of the agricultural systemand critically discriminate between unwanted weeds with vegetables andfruits to be harvested over vast areas.

The procedure used in the preferred embodiment to identify weeds 3 isshown with reference to FIGS. 3, 8 and 13 . Said mulch garden 5 is to besearched for weeds 3 by robot following a linear trajectory in autopilotmode S205 by reflective randomization. Detecting S220 said weed 3, andpositioning S225, S230, said robot 15 for extraction S235 of said weed 3requires machine vision weed detector 42 which includes weed machinevision camera 35 to provide input of visual information to CPU 100 forcommand and control of said robot 15. A person of ordinary skill in theart would recognize that there are many methods by which machine visioncan accomplish this task. However, the preferred embodiment of thisinvention utilizes a very simple and efficient method that is veryamenable to programming robots in real time with minimal computationalresources.

In the preferred embodiment, the machine vision weed detector 42, whichincludes a weed machine vision camera 35, seeks a green object withinthe rectangular field of view 370, shown in FIGS. 12, 14 (a), and 15(b).In the exemplar embodiment shown in FIG. 14(a), said rectangular fieldof view 370 is horizontal with a width, W 405, of 6 inches and length, L400, of 4 inches and is focused on the ground ahead of the robot 15 by adistance D 430, of 12 inches. This 6-inch by 4-inch field of view 370has an image of a weed 3 in mulch garden 5 shown in FIG. 14(a) underhigh resolution conditions (300 px/in) and in FIG. 14(b) under lowresolution conditions (5 px/in) as captured by machine vision camera 35and processed. The longitudinal-vertical plane passing through said lensof said machine vision camera 35 mounted on the longitudinal axis 435 ofsaid robot 15, as shown in FIGS. 12 and 15 (b), also passes through thecenter of said field of view 370. The intersection of this verticalplane with the horizontal plane of said field of view 370 defines alongitudinal axis of robot 435, shown in FIGS. 12 and 15 (b). As seen inFIG. 14(a), the Y-axis 375 of said camera field of view 370 is alignedwith the edge of said camera field of view 370 and parallel to saidlongitudinal axis of robot 435. The X-axis of said field of view 370 isaligned with its base and is perpendicular to said Y-axis 375, as seenin FIGS. 14(a), 15(a), and 15(b), which is also aligned with andparallel to a longitudinal axis of said robot 15 and which passesthrough the center of said lens of said machine vision camera 35. Atransverse-vertical plane passing through said center of said field ofview 370 intersects with said horizontal plane of said field of view370, shown in FIG. 14 , defines an X-axis 380 which is perpendicular tosaid Y-axis and which passes through the base of said field of view 370,as shown. Thus, an X-Y coordinate system having an origin at the forwardright corner of said field of view 370, as best seen in FIG. 15(b), andlongitudinally aligned with said vision camera 35 and the weed grabberclaws 45 is established. These coordinate axes are also aligned with thesquare pixels that define the image created by said machine visioncamera 35 and seen in FIG. 14(b) for the 5 pixel/inch case. In thisembodiment, the 6-inch by 4-inch field of view 370 is also characterizedby a 30 pixel by 20-pixel array that corresponds with the coordinateaxes. Said 30 pixel by 20-pixel array has 600 pixels in total, each onehaving a corresponding coordinate axis, X 380 and Y 375, respectively.This image is processed by said machine vision weed detector 42 as a600-element pixel array and is analyzed by CPU 100. In otherembodiments, a person of ordinary skill in the art may select adifferent resolutions and different fields of view 370, and thereby usean array of different number and dimension.

In the preferred embodiment, the RGB system of color synthesis is usedwhereby the color of each pixel is coded by three digital numberscorresponding to Red, Blue, and Green, each having values of N=0-250 inthe form (N_(R), N_(B), N_(G)). By way of example, the RGB code for acommon shade of dark brown is (78,67,63); for violet (238, 130, 238);for light golden brown (167, 133, 106) for forest green (34,139, 34);and for basic green (0,128,0). The RGB code for black is (0,0,0) and forwhite (255,255,255). The codes can also be expressed in hexadecimalform.

In the preferred embodiment, weeds are identified by virtue of thecontrast between the green weed and the dark mulch. By reducing theresolution to a low value of around 5 pixels per inch or less for lowresolution imaging, the pixel array inside the 6-inch by 4-inch field ofview 370 has a relatively small number of elements (600 in thisexample). Given the high clock speed of the CPU 100, in excess of 1.5GHz, calculations can be made in real time to ascertain the presence ofa weed 3 as the robot 15 moves linearly at a low speed through the mulchgarden 5 of between 5 to 20 feet per minute. When the presence of a weed3 is determined by the CPU 100, S220 in FIG. 8 , the robot 15 is haltedimmediately and then positioned for weed extraction S230.

In the preferred embodiment, the machine vision detector 42 has threecriteria to identify a weed 3. As said robot 15 follows its lineartrajectory in the autopilot mode, S205, the machine vision weed detector42 continuously searches for weeds 3 within the camera field of view370.

The first criterion is that the weed must be predominantly green. TheCPU 100 polls the RGB color numbers in the pixel array which is providedby the weed machine vision weed detector 42 in all 600 pixels within thearray and calculating, for each pixel, the fraction of G (green) asfollows:

${GF} = \frac{N_{G}}{N_{R} + N_{G} + N_{B}}$

where GF=Green Fraction

-   -   If GF≥0.40 save RGB color data    -   If GF<0.40 set RGB pixel data to (0,0,0) (black)

Thus, the pixel array is reconstituted such that only those pixels thatmeet criterion #1, GF>0.40, are identified with their actual RGBnumbers, while the remainder of the pixels are blackened by giving theRGB numbers (0,0,0). In FIG. 14(c) is shown the result of blackening thebackground RGB data for the case of a weed 3 as shown in FIG. 14(a). Theprocess is shown in FIG. 16 . Thus, according to the criterion #1, theCPU 100 must poll the RGB pixel array. If there are at least one pixelthat meets the GF≥0.40 criteria, the robot proceeds to criterion #2. Ifcriterion #2 is not met, said robot follows S205 whereby robot 15follows a linear trajectory in the autopilot mode.

The criterion #2 is whether or not the number of pixels meeting thefirst criterion are sufficient to constitute an actual weed 3 of aspecified size. The size can be determined by counting the number ofnon-black pixels. Each pixel is square and has an associated dimension,e.g., 0.2 inch for a resolution of 5 pixel/inch. Since we seek a weedlarger than 0.5 inches in linear dimension, corresponding to an areagreater than 0.25 square inches, this would correspond to about 6.25pixels meeting criterion 1. This criterion would be the minimum numberof Green pixels in the field of view 370 that would be identified as aweed 3. In the example shown in FIG. 14(c), there is about 25 pixels.Applying criterion 2, if there are less than 6.25 pixels with colorremaining in the array, the entire array is rejected and the robot 15continues its search in accordance with S205. If criterion 2 is met, anoptional third criterion of pixel connectivity is determined.

An optional criterion #3 is whether or not the pixels in the backgrounderased array are consolidated into a single weed entity, or are theyspread out in multiple locations around the camera field of view 370.This can be done by subdividing the camera field of view 370 into blocksand determining the green pixel density (GPD) within each block. Eachblock is then polled to determine the GPD:

${GPD} = {\frac{{{No}.{of}}{Green}{Pixels}{in}{block}}{{Total}{{No}.{of}}{Pixels}{in}{Block}} = \frac{N_{G}}{N_{T}}}$

where: Block=A preferably square subdivision of Field of View

-   -   N_(G)=Number of Green pixels in block.    -   N_(T)=Total Number of pixels in block

In the case of the exemplar preferred embodiment, the field of view 370is 6″×4″, the user can define a block as 2″×2″ subdivision, which wouldresult in six blocks of 25 pixels each, for a 5 px/inch resolution. CPU100 would calculate the GPD in each of the six blocks and poll them anddetermine if any one of the six blocks had a GPD≥GPD_(base), whereGPD_(base) is set by the user to determine the cohesiveness of thepixels. For the exemplary preferred embodiment, a preferred value isdetermined by the expected size of a weed. In the exemplary preferredembodiment:

${GPD_{base}} = {\frac{5}{25} = {{0.2}0}}$

Applying criterion 3, if after polling all of the six blocks, there isno block that has a GPD exceeding 0.2 the entire array is rejected andthe robot 15 continues its search in accordance with S205. If criterion3 is met, i.e., there is at least one block of the six with a GPD>0.2,the robot 15 halts its trajectory and control proceeds to weed machinevision positioning mode S225.

It is noted that the third criterion is optional because it is expectedthat the process will miss a certain fraction of weeds and that machinevision weed detector 42 will make some false identifications of weed 3.Criteria #3 helps to reduce the frequency of such errors. If criterion#3 is satisfied, CPU 100 determines that a weed has been detected inaccordance with S220.

As previously stated, once the machine vision weed detector 42identifies a weed 3 and transfers the relevant data to CPU 100, therobot 15 is halted and robot 15 follows weed machine vision positioningmode S225 and on to S230 to position weed grabber 45 directly abovedetected weed 3 as shown in FIG. 16 .

It is important to note that the collision avoidance system will notpermit a structure larger than a weed, such as a desirable plant orornament, to enter the field of view of the camera. Therefore, confusionof weeds with other objects is avoided.

In this embodiment, as best seen in FIGS. 15(a) and 15(b), the CPU 100commands robot 15 to position weed grabber 45 directly above thecentroid 385 of the projection of weed 3 in the camera field of view 370as required by S230. We therefore need to relate the coordinate systemof the robot 15 with the coordinate system of the camera field of view370. This is shown in FIGS. 15(a) and 15(b). In 15(a), we see the camerafield of view 370, also shown in FIG. 14(c), which has thelow-resolution view of the weed 3 with a blackened background, but withthe relevant coordinates identified. The centroid 385 of the weed 3 isshown along with its radial and angular coordinates (r_(Centroid), θ),with origin X=0, Y=0. The location of the centroid with respect to thecoordinates of field of view 370 can be calculated as follows:

${X_{C} = {\frac{\sum\limits_{1}^{N}{x_{i}\Delta_{i}}}{\sum\limits_{1}^{N}\Delta_{i}} = \frac{\sum\limits_{1}^{N}x_{i}}{N}}};{Y_{C} = {\frac{\sum\limits_{1}^{N}{y_{i}\Delta_{i}}}{\sum\limits_{1}^{N}\Delta_{i}} = \frac{\sum\limits_{1}^{N}y_{i}}{N}}}$${r_{C} = {r_{Centroid} = \sqrt{X_{C}^{2} + Y_{C}^{2}}}};{\theta_{C} = {\tan^{- 1}\left( \frac{Y_{C}}{X_{C}} \right)}}$

where: (X_(C), Y_(C)) are the Cartesian position coordinates of theCentroid of Weed 3

-   -   (r_(C), θ_(C)) are the cylindrical position coordinates of the        Centroid of Weed 3    -   (x_(i), y_(i)) are the Cartesian position coordinates of the        i^(th) pixel of weed 3 which correspond to the RGB pixels with        G≠0.    -   Δ_(i)=area of i^(th) pixel=Δ=common area for all pixels    -   N=Total number of pixels of weed 3 (total number pixels with        G≠0)

As shown in FIG. 15(b), once the coordinates (r_(C), θ) of the centroid385 of the weed 3 is determined relative to the X-Y coordinates of thefield of view 370, the position coordinates (R,ϕ) of the weed 3 relativeto the center of the weed grabber claws 45 must be determined. As seenin FIG. 15(b), the coordinates (R,ϕ) can be determined as follows:

$R = \sqrt{\left( {{r_{C}\cos\theta} - {W/2}} \right)^{2} + \left( {{r_{C}\sin\theta} + D} \right)^{2}}$${\tan\phi} = \frac{{r_{C}\cos\theta} - {W/2}}{{r_{C}\sin\theta} + D}$

Once the coordinates (R,ϕ) are determined, CPU 100 can then commandrobot 15 to position said weed grabber claws 45 directly above detectedweed 3 in accordance with S230. Thus, in accordance with FIG. 15(b),once the centroid of the weed 3 is determined, said robot 15 must changethe angular direction of its longitudinal axis 435 by an angle ϕ 415 andmove forward on a linear trajectory a distance R 410. At this point,said weed grabber claws 45 will conform to S230 and be ready to commenceS235 so as to activate the weed extraction module 130 in accordance withS240, S245, and S250.

The weed extraction module 130 comprises claw elevation servo 300 and aclaw grasping servo 305. FIG. 10 shows the operation of said servos 300and 305. Claw elevation servo 300 is fixedly mounted on the base 95 ofsaid robot 15 and output shaft of said claw elevation servo 300 is fixedto a servo frame 325 upon which said weed grabber claws 45 are mounted.As best seen in FIG. 10 steps S350 and S360, in accordance with commandsfrom CPU 100, claw elevation servo 300 can retract said weed grabberclaws 45 as seen in S350, or it can engage said weed grabber claws 45 asseen in S360.

Said grasping claw servo 305 is fixedly mounted on said servo frame 325upon which servo frame claw 315 is integrally included, or fixedlyattached. Grasping claw 310 is fixed to the pivoting output shaft ofsaid claw grasping servo 305 in such a manner as when grasping clawservo 305 is so commanded by CPU 100, grasping claw 310 rotates withsaid output shaft of claw grasping servo 305 so as to pivot towardsservo frame claw 315 as seen in FIG. 10 , S330, where claws can grasp aweed, or pivot away from said servo frame claw 315 as seen in S340 so asto open said claws 45.

In operation, as seen in FIGS. 8 and 9 , after said robot 15 ispositioned for weed removal S230, said claw elevation servo 300 isactivated to cause open claws 45 to descend and engage the groundsurrounding a weed S235, then claw grasping servo 305 causes jaws ofgrabber to close around weed and hold it firmly S240. Weed grabber clawelevation servo 300 is then commanded by CPU 100 to pull said weedvertically away from ground S245. When claws 45 are in their upwardposition, CPU 100 commands claw grasping servo 305 to open while alsocommanding suction tube to clean claws 45 and transport said weed toweed receptacle 55, S250. CPU 100 then returns control to robotnavigation module 110 and robot 15 continues searching for weed on alinear trajectory, S205.

While the preferred embodiment is the use of grabber claws 45 in theweed extraction module 130, as described to remove weeds efficiently ina mulch garden 5 and shown in FIG. 7 , this invention does anticipatethe use of other manners of weed extractors such as screw augers,scrapers, electric arc, lasers, and sprays of herbicides, and otherknown means which can replace the grabber claws 45 and follow stepscorresponding to S235 through S250 in FIG. 8 , with certain programmingmodifications well known to persons of ordinary skill in the art.

As seen in FIGS. 4, 5, and 6, and 7 , the propulsion system of thepreferred embodiment of the robot 15 consists of a right front drivewheel 60 powered by electric drive motor for right front wheel 80 and aleft front drive wheel 65 powered by electric drive motor for left frontwheel 85 which are independently controlled by CPU 100 through motorcontroller 82. Both rear wheels 70 and 75 are coaxially driven by asingle drive motor for rear wheels 90 which are also independentlycontrolled by CPU 100 through motor controller 82.

As seen in FIG. 7 , the CPU 100 controls the direction of said robot 15by sending commands to said propulsion module 120, which causes thefront wheels 70 and 75 to rotate at different speeds, either in the samedirection of rotation, or in opposite directions of rotation. If needed,this arrangement permits said robot 15 to pivot angularly about avertical central axis without appreciable forward motion so that forcollision avoidance, for example, said robot 15 moving along its lineartrajectory can stop, pivot, and continue on a linear trajectory ifcommanded by CPU 100. Such an outcome might occur if CPU 100 issuescommands in response to domain boundary detector 125 which detects aboundary 12, collision avoidance detector 40 which detects an obstacle,or weed detector 35 which detects a weed.

CPU 100 can control the speed of said robot 15 and is programmable bythe user with a preferred speed between 5 and 20 feet/minute. Similarly,CPU 100 can brake said robot 15 by reducing the speeds of said wheeldrive motors 80, 85, and 90.

Said robot 15 must be small enough so that it can maneuver betweenshrubs and obstacles but be large enough so that it can position itselfover a weed and extract it. Furthermore, said robot 15 must carry anelectrical power supply if significant size and weight so as to powersaid drive motors 80, 85, and 90, said servos 300 and 305, as well aslighting and electrical components, and to so providing a sufficientperiod of operation to effect weed removal from said mulch garden 5.While no means critical or a limitation to this invention, for typicalmulch garden weeds in the size range of two inches, the preferredembodiment has a size for said robot 15 of a width of 9 to 18 inches,and a length of 9 to 18 inches, and a weight of 2.0 to 4.0 pounds. Theweight of said robot 15 must be sufficient prevent vertical movement ofsaid robot due to the vertical reaction forces caused by the penetrationforces of said weed grabber claws 45, or other extraction means, to thedepth of the weeds' roots. The weight of said robot 15 may be increasedby the addition of ballast weights (not shown). For grabber claws 45shown in a preferred embodiment, the vertical reaction force istypically less than two pounds. A user may wish to increase or decreasethe size and weight of said robot depending on special requirement ofparticular applications.

A user may wish to use a larger or smaller battery, depending on thetime needed to extract weeds from said mulch garden 5 and on the size ofsaid robot.

Those skilled in the art will readily recognize numerous adaptations andmodifications which can be made to the present invention which willresult in an improved system and method for removing weeds from anaesthetic mulch garden using an autonomous robot, yet all of which willfall within the spirit and scope of the present invention as defined inthe following claims. Accordingly, the invention is to be limited onlyby the scope of the following claims and their equivalents.

What is claimed is:
 1. A weed detecting and weed removal apparatus forbounded aesthetic mulch gardens comprising: an autonomous terrestriallymobile robot including a propulsion module for controlling thetranslational motion of said robot; and, a programmable centralprocessing unit for issuing positioning commands to said robot andissuing commands to servo motors, and receiving inputs from a pluralityof sensors, and performing calculations incorporating input from saidsensors, to generate said commands; and, a domain boundary detectionsystem to define the outside boundary of said mulch garden and confinethe trajectory of said robot to the domain within said boundary; and, acollision avoidance system for preventing interactions between saidrobot and obstacles present within said aesthetic mulch garden; and, amachine vision weed detector for capturing images of a defined field ofview within said mulch garden and identifying the presence of one ormore weeds; and, a weed extraction module including one or more servomotors for removing said weeds from said mulch garden; and, anelectrical power supply mounted on said robot to provide power to saidpropulsion module and to energize said central processing unit, saidsensors, said machine vision weed detector, and said weed extractionmodule.
 2. The apparatus of claim 1 further comprising: a propulsionmodule having a plurality of wheels which are driven by independentdrive motors each of which are electrically connected to a motorcontroller which receives commands from said central processing unit todifferentially actuate each wheel and thereby control direction andspeed of said robot.
 3. The apparatus of claim 1 having said propulsionmodule directed by said central processing unit to have said robotfollow an essentially linear trajectory at a predefined speed unlesssaid central processing unit receives a command from either said domainboundary detection system detecting proximity to said boundary, or saidcollision avoidance system detecting proximity to an obstacle, whichresults in a command to change direction by a user-prescribed angle andcontinue on said linear trajectory.
 4. The apparatus of claim 1 havingsaid propulsion module directed by said central processing unit to havesaid robot follow an essentially linear trajectory at a constant speedunless said central processing unit receives a command from said machinevision weed detector upon which said central processing unit wouldcommand said robot to halt for calculations to determine if a weed hasbeen encountered and, if said weed is encountered, to command said robotto assume position for weed extraction; or, if said weed was notencountered, directing said robot to proceed with an essentially lineartrajectory.
 5. The apparatus of claim 1 continuing to seek and removesaid weeds until 1) a timer in said central processing unit has reacheda preprogrammed time set by the user; or, 2) weeds are not encounteredfor an interval of time programmed by the user; or 3) said centralprocessing unit detects a failure in any of said robot's systems; or, 4)the user externally intervenes and shuts power to said robot.
 6. Theapparatus of claim 1 further comprising: said boundary of said mulchgarden defined by a plurality of magnetic stakes spaced at predeterminedintervals and surrounding said boundary of said mulch garden, thespacing calculated to maintain a magnetic field strength which can bedetected by a magnetometer mounted on said robot which sends a signal tosaid programmable central processing unit.
 7. The domain boundary systemof claim 6 further comprising: magnetic stakes each with a magnetmounted thereon selected from a group consisting of 1) a neodymiumpermanent magnet; 2) a ceramic ferrite permanent magnet; 3) a rare earthpermanent magnet; and 4) a direct current electromagnet.
 8. Theapparatus of claim 7 where the polarization axis of said magnet isvertical and parallel to the longitudinal axis of said stakes.
 9. Theapparatus of claim 1 further comprising: a said domain boundarydetection system defining the boundaries of said mulch garden selectedfrom a group of sensing apparatuses consisting of 1) said boundary ofsaid mulch garden defined by a below-ground electric cable whichtransmits radio waves to define said boundary and a radio receivermounted on said robot; 2) said boundary of said mulch garden defined bylongitudinal and latitudinal position coordinates specified by the userwhich are compared with a global positioning system (GPS) detectormounted on said robot and interacting with said central processing unitto compare current position of said robot with said GPS coordinates; and3) said boundary of said mulch garden defined by a light emitting dioderope placed on the boundary of said mulch garden, and a photodetectorplaced on said robot and communicating with said central processing unitto determine location of said robot relative to said boundary.
 10. Theapparatus of claim 1 further comprising: said collision avoidance systemselected from a group consisting of: 1) an ultrasonic range sensor; 2) aradar range sensor; and, 3) a laser range sensor.
 11. The apparatus ofclaim 1 further comprising: Said machine vision weed detector having avision sensor mounted on the front of said robot and focused on theground with a field of view located at a fixed distance ahead of saidrobot; said vision sensor capable of transmitting color data to themachine vision weed detector which thereupon conveys said color data tosaid central processing unit for calculations and analysis.
 12. Theapparatus of claim 11 wherein said vision sensor is a camera.
 13. Theapparatus of claim 11 wherein said field of view is illuminated by alamp mounted on said robot.
 14. The apparatus of claim 11 wherein saidcolor data is digitized by a color system selected from a groupconsisting of: 1) RGB; 2) CMY; and 3) CMYK color systems.
 15. Theapparatus of claim 14 having said central processing unit identifyingthe presence of one or a plurality of weeds by reducing the resolutionof the images in said defined field of view by calculating the GreenFraction of each of the pixels, blackening all pixels having a GreenFraction below a user specified value, counting the number of higherGreen Fraction pixels and determining if they exceed a user specifiednumber characteristic of the size of a weed, and calculating thelocation of the centroid of said higher Green Fraction pixels toidentify position of said weed relative to said robot; and, issuingcommands to said propulsion module to locate said robot in a positionfor extraction of said weeds.
 16. The apparatus of claim 15 furtherdetermining the density of higher Green Fraction pixels within saidfield of view as a further criterion for determining if said weed ispresent by subdividing the field of view into blocks and instructingsaid central processing unit to calculate the Green Pixel Density withineach block, and determining if any one of the reduced number of blockshad a Green Pixel Density greater than a user specified criterion forthe presence of a weed which is representative of the type and size ofthe weeds being sought.
 17. The apparatus of claim 1 wherein said weedextraction module for removing said weeds from said mulch garden isselected from a group consisting of: 1) grabber claws; 2) screw augers;3) scrapers; 4) electric arcs; 5) lasers; and 6) sprays of herbicides.18. The apparatus of claim 17 where said central processing units sendscommands to said grabber claws to elevate and engage said grabber clawswith the weed, and to open and close said grabber claws which areenergized by two respective servo motors.
 19. The apparatus of claim 1where said weeds removed from said mulch garden are transported fromsaid weed extraction module to a receptacle by means of a suction tubeenergized by a blower and activated by said central processing unit. 20.A method for weed detecting and weed removal for bounded aesthetic mulchgardens comprising: activating an autonomous terrestrially mobile robotincluding a propulsion module for controlling the translational motionof said robot; and, electrically connecting and mounting to saidprogrammable central processing unit for issuing positioning commands tosaid robot, issuing commands for weed extraction, and receiving inputsfrom a plurality of sensors, and performing calculations to generatesaid commands; and, providing an electrical power supply mounted on saidrobot to provide power to said propulsion module and to energize saidcentral processing unit and said sensors; and, providing a domainboundary detection system to define the outside boundary of said mulchgarden and confine the trajectory of said robot to within said boundary;and, providing a collision avoidance system for preventing interactionsbetween said robot and obstacles present within said aesthetic mulchgarden; and, providing a machine vision weed detector for capturingimages of a defined field of view within said mulch garden andidentifying the presence of one or a plurality of weeds; and, providinga weed extraction module for removing said weeds from said mulch garden.